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Summary of Results: Crawford and Raia (1986) found that students in the intervention group 

scored significantly higher than students in the comparison group on math 
computation, but not on total math or math concepts. Since the analysis 
was done at the student level while the intervention assignment was at the 
classroom level, caution should be used in interpreting the results. 
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The What Works Clearinghouse (www.whatworks.ed.gov) was established in 2002 by the U.S. 
Department of Education’s Institute of Education Sciences to provide educators, policymakers, 
researchers, and the public with a central and trusted source of scientific evidence of what works in 
education. Please email all questions and comments to info@whatworks.ed.gov . The What Works 
Clearinghouse is administered by the U.S. Department of Education through a contract to a joint 
venture of the American Institutes for Research and the Campbell Collaboration . 
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Intervention: Saxon Math 

Operational Features 

Saxon Math is a series of mathematics curricula 
designed for use in kindergarten through grade 
12. According to the developers, the Saxon 
philosophy for teaching mathematics, 
irrespective of grade, is that children develop 
and retain mathematics skills best when 
curricula are based on four principles: (1) 
instruction is incremental and distributed across 
each grade level; (2) students have the 
opportunity for continual practice across each 
grade level; (3) assessments of student 
knowledge are cumulative and occur frequently 
across each grade level; and (4) mathematics is 
taught via direct instruction. The Saxon 
textbooks break up all concepts into increments 
(i.e., small, understandable pieces) and the 
content covered within and across each grade 
builds upon previously learned concepts. 

Topics are never dropped, but they increase in 
complexity. 

According to the developers, Saxon Math covers 
all five content and skill areas of the National 
Council of Teachers of Mathematics (NCTM) 
standards, and the content and skills also meet 
various state standards. Each grade of Saxon 
Math consists of 120 daily lessons and 12 
activity-based investigations. Teachers 
facilitate each daily lesson, which consists of a 
warm-up, introduction to the new concept, 
practice focusing on the new concept, and 
practice focusing on both the new and 
previously learned concepts. Students are 
assessed cumulatively, every fifth lesson, and 
must master a lesson at 80% or better before 
moving on to the next lesson. A number of 
materials are available at each grade level and, 
according to the developers, a number of 
supports are available to teachers of Saxon 
Math. 

In this study, students using Saxon Algebra 1/2 
were compared to students using the Scott- 
Foresman Mathematics curriculum. 

People, Settings, and Timing 

Crawford and Raia (1986) conducted three 
studies with different subsets of the overall 



sample. This study report focuses on the 
strongest because it used a matched sample 
only, which accounts for differences between 
schools and teachers. The target population in 
the third study consisted of four teachers within 
four pilot middle schools in the Oklahoma City 
Public Schools (OCPS). All four teachers 
taught both the intervention (Saxon) and 
comparison (Scott-Foresman) texts. All 
students were from the 8th grade. Students in 
both the intervention and comparison groups 
were assessed on the outcome measure at the 
end of the academic year in 1985. 

Cost Information 

Crawford and Raia do not provide cost 
information for implementing Saxon Math in the 
OCPS. However, extensive cost information for 
the Saxon math curricula and related materials 
is available on the developers’ website 
(www.saxonpublishers.com) . 

Intended Duration 

Saxon Math is a curriculum that consists of 120 
daily lessons. However, Crawford and Raia do 
not indicate how many lessons students in the 
Saxon Math classrooms completed and how 
many lessons need to be completed for the 
curriculum to be implemented as intended. The 
curriculum was piloted in the OCPS middle 
schools during the 1984-1985 academic year. 

Scientific Rationale 

Crawford and Raia offer no scientific rationale 
for conducting this evaluation. However, they 
state that OCPS had already evaluated two 
previous Saxon curricula ( Saxon Algebra 1 and 
Saxon Algebra 1 1/2). The studies of Saxon 
Algebra 1 concluded that students who 
participated in the Saxon classrooms scored 
significantly higher than students in the “status 
quo” classrooms on two algebra outcome 
measures, so OCPS adopted the Saxon Algebra 
1 text. The study of Saxon Algebra 1 1/2 found 
no statistically significant effect of the Saxon 
text over the status quo, and the evaluator 
recommended that OCPS not adopt the Saxon 
Algebra 1 1/2 text. As stated by Crawford and 
Raia, when Saxon introduced a third textbook 
(Saxon Algebra 1/2), OCPS wanted to conduct a 
pilot study to determine its effectiveness since 
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the previous studies of the Saxon textbooks had 
produced mixed results. The study was needed 
to address the school district’s interest in 
determining what educational treatments 
enhance student achievement. 

Overview of the Study 

Purpose 

This evaluation was designed to compare 
student achievement scores of students enrolled 
in Saxon Algebra 1/2 classrooms (intervention 
group) with students enrolled in Scott-Foresman 
Mathematics classrooms (comparison group). A 
secondary purpose was to compare the textbook 
content of Saxon Algebra 1/2 and Scott- 
Foresman Mathematics. The WWC only 
reviews evidence in relation to achievement; 
therefore, the comparison of textbook content is 
not reviewed further in this study report. As 
stated above, Crawford and Raia conducted 
three analyses, but this study report focuses on 
the strongest analysis of the three and describes 
how Saxon Algebra 1/2 compared to the 
comparison text only for those teachers who 
taught both textbooks in a sample of students 
matched on pretest California Achievement Test 
(CAT) total math scores. 

Intervention Fidelity 

As described by the curriculum developers, 
Saxon Algebra 1/2 is intended to be a full-year 
program of instruction, lessons are designed to 
be one hour daily (this includes practice and 
review time), and assessments are to occur 
every fifth lesson, usually on each Friday of the 
week. Crawford and Raia do not provide any 
information about the implementation of the 
curriculum or teacher training. Teachers do 
have access to various supports and resources 
through the Saxon Math website and telephone 
support. 

Outcome Measures 

The outcome measure was the CAT total math, 
math computation, and math concepts scores. 
The CAT is a nationally normed, valid, and 
reliable test designed to measure achievement in 



the basic skills taught in school. The outcome 
measure was administered at the end of the 
academic year in 1985 and appears to be well 
aligned to the curriculum. 

Research Design 

The research design of study three was quasi- 
experimental, and this study report only reviews 
the evidence related to the stratified matched 
sample within the third study. It is not clear 
how Crawford and Raia selected the overall 
sample for the study or how the sample was 
placed in groups except that it appears to be 
somewhat of a convenience sample (some of the 
teachers used the intervention text and some of 
them used the comparison text). For study 
three, Crawford and Raia focused only on those 
teachers who taught both the intervention and 
comparison textbooks. Because there were 
pretest CAT total math achievement differences 
between students, Crawford and Raia used 
stratified matching to select a sample of students 
who had exactly the same mean on pretest total 
math scores for the teachers that taught both the 
intervention and comparison texts. 

Participant Flow 

Crawford and Raia assigned participants to the 
intervention and comparison groups at the level 
of the class. At the beginning of study three, 
four teachers taught both the Saxon Algebra 1/2 
and the Scott-Foresman Mathematics curricula. 
The number of 8th-grade students in the 
intervention group was 72 and in the 
comparison group was 259. For this study 
report, the matched sample at the beginning of 
study three consisted of four teachers who 
taught both curricula. The number of 8th-grade 
students in the matched sample was 39 in the 
intervention group and 39 in the comparison 
group. 

Reference Periods 

Crawford and Raia piloted Saxon Algebra 1/2 in 
the OCPS middle schools during the 1984-1985 
academic year. The CAT was administered at 
the end of the 1985 academic year. 
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Baseline Data 

Crawford and Raia did not describe baseline 
characteristics for the intervention and 
comparison groups except for the grade level of 
the students. Crawford and Raia also pretested 
the students on the CAT and used the CAT total 
math score as a pretest to match a subset of the 
third study sample. (See Table 1.) 

There were significant pretest differences 
between groups and Crawford and Raia used 
matching on total math pretest scores to control 
for these pretest differences in the analyses of 
interest for this study report. 

Statistical Methods 

Crawford and Raia conducted the analysis of the 
matched sample in study three at the student 
level, representing a mismatch between unit of 
assignment and unit of analysis. In study three, 
Crawford and Raia performed a number of 
analyses. Crawford and Raia utilized analyses 
of variance (ANOVAs) to compare total math, 
math computation, and math concepts scores 
between the matched intervention and 
comparison groups. Crawford and Raia present 
pretest, posttest, and gain score means, standard 



deviations, and F statistics for each of the 
analyses, but only the posttest scores are 
presented in Table 2 and analyses of posttest 
outcomes scores are reported in the next section. 

Outcomes and Estimation 

Table 2 and Figures la-c present the effects of 
the Saxon Algebra 1/2 curriculum on CAT total 
math, math computation, and math concepts 
scores, as reported by Crawford and Raia. The 
ANOVA analyses indicate that students in the 
intervention group scored significantly higher 
on one of three measures, math computation, 
than students in the comparison group. There 
were no significant differences between the 
intervention and comparison groups in the other 
measures, posttest math concepts and total math 
scores. 

Cautions 

The sample sizes for the matched sample in 
study three were small, thereby making 
inferences problematic. Since the analysis was 
done at the student level while the level of 
intervention assignment was done at the 
classroom level, caution should be used in 
interpreting the results. 
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Table 1. Pretest Characteristics of the Study Sample (Study Three 
Matched Sample Only) 





Intervention 
(n = 39) 


Comparison 
(n = 39) 


Grade 8 


39 


39 


Pretest CAT Total Math Score 


51.7 


51.7 


Pretest CAT Math Computation Score 


50.6 


50.8 


Pretest CAT Math Concepts Score 


52.3 


51.8 



Note. Pretest characteristics are reported at the student level (this is the unit of analysis utilized by Crawford and Raia) for 
the matched sample within study three only (this is the only relevant analysis for the WWC study report). Crawford and 
Raia assigned to the intervention and comparison groups at the level of the class. CAT = California Achievement Test. 



Table 2. Impact Reported by Crawford and Raia (1986): CAT Total Math, Math 
Computation, and Math Concepts Posttest Scores 





Intervention 


Comparison 




(n = 39) 


(n = 39) 


Total Math 






Mean (SD) 


55.6(11.9) 


50.7(11.8) 


Sample Size 


39 


39 


Math Computation 

Mean (SD) 


57.6(13.4) 


51.5 (14.1) 


Sample Size 


39 


39 


Math Concepts 

Mean (SD) 


53.4(12.4) 


49.8 (12.4) 


Sample Size 


39 


39 



Note. These are posttest score results for the CAT total math, math computation, and math concepts tests calculated 
at the level of the student by Crawford and Raia (1986). CAT = California Achievement Test. 
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Figure la. Impact Calculated by Crawford and Raia (1986): CAT Total 
Math Posttest Scores 
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Note. The analyses are done at the level of the student, which does not match the unit of 
assignment (class). Crawford and Raia (1986) reported that the intervention group posttest scores 
did not differ significantly from the comparison group posttest scores (p = ns). CAT = California 
Achievement Test. 
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Figure 1b. Impact Calculated by Crawford and Raia (1986): CAT Math 
Computation Posttest Scores 
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Note. The analyses are done at the level of the student, which does not match the unit of 
assignment (class). Crawford and Raia (1986) reported that the intervention group posttest scores 
were significantly higher than the comparison group posttest scores (p = .05). CAT = California 
Achievement Test. 
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Figure 1c. Impact Calculated by Crawford and Raia (1986): CAT Math 
Concepts Posttest Scores 
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Note. The analyses are done at the level of the student, which does not match the unit of 
assignment (class). Crawford and Raia (1986) reported that the intervention group scores did not 
differ significantly from the comparison group scores (p = ns). CAT = California Achievement Test. 



info @ saxonpublishers .com 
www.saxonpublishers.com 

Related Studies 

See reports on other studies of Saxon Math . 

See reports on other studies of Middle School 
Math curricula. 
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Intervention Developer Contact 
Information 



Report Production 

Date created: March 4, 2005 

Topic area reviewed under: Curriculum-Based 
Interventions for Increasing K-12 Math 
Achievement — Middle School 



Saxon Publishers 
2600 John Saxon Blvd. 
Norman, OK 73071 
(800) 284-7019 




WWC Study Ratings 3 : Crawford and Raia (1986) 



Causal Validity: 


Meets WWC Evidence Standards, a Quasi-Experimental Design with Stratified 
Matching 


Students in the intervention group ( Saxon Algebra 1/2) were compared to students in the comparison group (Scott-Foresman 
Mathematics). It is not clear how teachers and students were identified for the study or how they were placed in groups other 
than that classes that implemented Saxon Algebra 1/2 seemed to be a convenience sample. This study report focuses on the 
matched sample within study three. Crawford and Raia matched the students who were in the classes where the teachers 
taught both curricula on pretest California Achievement Test (CAT) total math scores and compared the posttest total math, 
math computation, and math concepts scores between groups. There does not appear to be attrition at the level of assignment 
or analysis within this subsample. Crawford and Raia do not report any extraneous events that might have confounded the 
intervention's effects. 


Other Study 
Characteristics 


Study Rating Study-Specific Information 


Intervention Fidelity 


• 


Saxon Math meets the definition of Middle School Math. However, Crawford and Raia 
do not provide information about the implementation of the program. There was no 
measure of teacher implementation, and teacher training was not discussed. 


Outcome Measures 


• • 


One achievement measure was reported in the study, the CAT, a test of basic skills 
including math computation and math concepts. The CAT is a nationally normed, 
standardized test, has acceptable reliability and validity, and appears to align to the 
intervention. 


People, Settings, and 
Timing 


• 


Although the sample of students is part of the identified middle school math 
population, it was homogeneous with respect to students (no subgroup variation) and 
settings. The outcome measure was implemented at a time appropriate to capture the 
effect of the intervention, but there was no variation in the type of achievement 






measure. 


Testing within 
Subgroups 


• 


The effect of Saxon Algebra 1/2 on students’ mathematics achievement was tested 
across the entire matched sample, but was not tested within subgroups of the sample or 
variations across settings. 


Analysis 


• 


The unit of assignment (class) was not the same as the unit of analysis (student). The 
statistical properties of the data allowed for valid estimates of effect sizes, but sample 
sizes were not adequate to provide sufficiently precise estimates of effect sizes. The 
sample sizes were small for the matched sample in this study, making inferences 
problematic. 


Statistical Reporting 


• • 


Sufficient data were reported, but they were reported at the level of the student. 
Crawford and Raia reported unadjusted means and standard deviations. Effect sizes for 
the outcome measure could be calculated using a standard formula if means and 
standard deviations were reported at the level of the teacher for the matched sample in 
study three. 


Note. • • Fully meets criteria; • Meets minimum criteria; X Does not meet criteria. 

a For more information on the criteria used to rate this study, see the WWC Evidence Standards: Middle-School Math. 
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